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ABSTRACT 



This report describes one of a series of pilot studies that 
were conducted to evaluate the instructional effectiveness of the Harcourt 
Math Program. Harcourt School Publishers (HSP) contracted with the 
Educational Research Institute of America (ERIA) to conduct a .series of 
independent pilot studies to evaluate the effectiveness of the Harcourt Math 
Program. HSP sought out volunteer teachers to participate in the study, the 
Harcourt Department of Test Services scored the standardized tests, and ERIA 
conducted the study and analyzed the data that were collected. The study 
described in this report was conducted during the spring of the 1999-2000 
school year. Research questions included: (1) Is the Harcourt Math Program 

instructionally effective?; and (2) Do selected chapters significantly 
increase students' understanding of key math skills, concepts, and strategies 
as measured by the program's chapter tests and by standardized achievement 
tests (i.e.; Stanford)? A total of three teachers from grades 2, 5, and 7 
volunteered to participate in the study. A quasi -experimental 
pretest-posttest design was used. Before instruction began, students were 
administered two pretests. The increase in test scores on the nationally 
standardized tests, the Stanford Achievement Tests, and on the Harcourt Math 
Assessment were both positive and statistically significant for all subtests 
and total test scores at grades 2, 5, and 7. (ASK) 
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A Study of the Instructional Effectiveness 
of the Harcourt Math Program 



by Paul Lloyd 

Educational Research Institute of America 

This report describes one of a series of pilot studies that were conducted to evaluate the 
instructional effectiveness of the Harcourt Math Program. 

Background Information 

Harcourt School Publishers contracted with the Educational Research Institute of America 
(ERIA) to conduct a series of independent pilot studies to evaluate the effectiveness of the 
Harcourt Math Program. Harcourt School Publishers sought out volunteer teachers to 
participate in the study; the Harcourt Department of Test Services scored the standardized tests; 
and ERIA conducted the study and analyzed the data that were collected. The study described in 
this report was conducted in the spring of the 1999-2000 school year. 

Research Questions 

The following research questions guided the design, procedures, and data analysis of the pilot 
study: 

Is the Harcourt Math Program instructionally effective ? Do selected chapters 
significantly increase students’ understanding of key math skills, concepts, and strategies 
as measured by the program’s chapter tests? As measured by standardized achievement 
tests (Stanford)? 

Design and Procedures of the Study 

This study included grades 2, 5 and 7. 

Teachers who volunteered to participate in the study had not used the program previously. The 
teachers were encouraged to select one cluster of chapters to pilot over a 6-week period. They 
also agreed to administer data collection instruments before beginning instruction and again after 
completing instruction. 

A total of 3 teachers volunteered to participate in the study: 1 each at grades 2, 5, and 7. The 
participating teachers came from 3 different schools in New Jersey. 

A quasi-experimental pretest-posttest design was used. Before instruction began, students were 
administered two pretests. The classroom teachers administered all tests. Table 1 summarizes 
the data collection instruments that were used. 
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TABLE 1 

Data Collection Instruments 



Grade 


Pretests 


Posttests 


2 


• Harcourt Math Program Chapter Tests 

• Math Subtests of the Stanford 
Achievement Test, Ninth Edition, 

Level P2 


• HACOURT MATH Program Chapter 
Tests 

• Math Subtests of the Stanford 
Achievement Test, Ninth Edition, 
Level P2 


5 


• Harcourt Math Program Chapter Tests 

• Math Subtest of the Stanford 
Achievement Test, Ninth Edition, Level 

12 


• Harcourt Math Program Chapter Tests 

• Math Subtest of the Stanford 
Achievement Test, Ninth Edition, Level 12 


7 


• Harcourt Math Program Chapter Tests 


• Harcourt Math Program Chapter Tests 



Teachers selected the cluster of chapters appropriate for their class. Following the pretests, 
teachers taught the selected chapters using the resources and procedures contained the Harcourt 
Math Program. Teachers also received training from experienced consultants in methods of 
implementing and using the program. While each cluster of chapters took 6 weeks to teach, most 
teachers started the pretests in February and completed the posttests in May. Instruction included 
a broad sample of math skills from the strands of problem solving skills, figuring change and 
time, using graphs and tables, understanding fractions, and study skills. 

Upon completion of the selected chapters, students were administered the posttests. All data 
collection instruments were returned to the Educational Research Institute of America where 
they were processed. The Stanford 9 Achievement Tests was scored at the Harcourt Educational 
Measurement Scoring Center. The Harcourt Math Program chapter tests were scored at ERIA, 
and all of the data were analyzed by ERIA. 




2 



Instructional Effectiveness of the Collections Reading/Language Arts Program 



Findings 

Results of the study and the descriptions of each of the assessments are reported in this section of 
the report. 

TABLE 2 

Learning Goals for Grade 2 Chapters 6-9 



Grade 2 Chapters 6-9 




To count on to identify amounts of money using coins 


To act out and solve problems by using coins 


To use coins to show amounts to .99 


To count coins and identify objects that can be bought with that amount 


To figure change by counting on with pennies 


To tell time to the hour, half-hour, 5 minutes, and 15 minutes 


To solve problems using elapsed time 


To read and use a calendar 


To use clocks and elapsed time to solve problems 


To sequence a series of events 


To use a schedule to solve problems 



Grade Two Test Results 



Table 1 summarizes the Pretest and Posttest means and standard deviations for the Grade 2 Math. 

TABLE 3 

Grade 2 Results (N=25) 



Grade 2 


Math (14 test items) 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest 


3 


14 


9.84 (70%) 


2.61 


Posttest 


10 


14 


12.75 (91%) 


1.26 



A paired t-test for the mean Harcourt Math Assessment showed that the scores improved 
significantly after instruction (t=6.87; p<.0001). 
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TABLE 4 

Grade 2 Results Stanford Achievement Test (N=23) 



Grade 2 


Stanfo 


rd Achieverrx 


snt Tests: Grade 2 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest: 

Procedures 


11 


20 


16.83 (84%) 


2.93 


Posttest: 

Procedures 


13 


20 


18.04 (90%) 


1.74 


Pretest: Problem 
Solving 


12 


28 


22.17 (74%) 


3.88 


Posttest: Problem 
Solving 


19 


30 


25.26 (84%) 


3.05 


Pretest: Total 
Math 


26 


47 


39.00 (78%) 


6.09 


Posttest: Total 
Math 


36 


49 


43.30 (87%) 


3.84 



A paired t-test for the two subtests and the total test score on the Stanford Achievement Tests: 
Grade 2 resulted in a significant increase (Procedures: r=2.48; pc.Ol); (Problem Solving: t- 4.75; 
p<.05); (Total Math: r=5.20; p<.0001). 



Grade Five Test Results 



TABLE 5 

Learning Goals for Grade 5 Chapters 15-18 



Grade 5 Chapters 15-18 




To identify, read, and write fractions 


To identify, read, and write mixed numbers and rename fractions greater than 1 as mixed numbers 


To compare fractions with unlike denominators 


To order fractions and draw a diagram to solve problems 


To find the greatest common factor of two numbers 


To find equivalent fractions 


To find the simplest form of a fraction 


To add fractions with like denominators 


To use least common denominator to add fractions with unlike denominators 


To subtract fractions with like denominators 


To use the least common denominator to subtract fractions with unlike denominators 


To subtract fractions of an inch on a ruler 
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Table 6 summarizes the Pretest and Posttest means and standard deviations for the Grade 5 Math. 

TABLE 6 

Grade 5 Results (N=46) 



Grade 5 


Math (20 test items) 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest 


1 


16 


6.30 (32%) 


2.56 


Posttest 


8 


20 


14.07 (70%) 


3.42 



A paired t-test for the mean Harcourt Math Assessment showed that the scores improved 
significantly after instruction (t=15.25; p<.05). 



TABLE 7 

Grade 5 Results Stanford Achievement Test (N=45) 



Grade 5 


Stanfc 


>rd Achievem 


tent Tests: Grade 5 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest: 

Procedures 


3 


19 


10.18 (51%) 


3.51 


Posttest: 

Procedures 


3 


20 


11.67 (58%) 


4.70 


Pretest: Problem 
Solving 


7 


27 


19.13 (64%) 


4.31 


Posttest: Problem 
Solving 


8 


30 


21.62 (72%) 


5.40 


Pretest: Total 
Math 


14 


46 


29.31 (59%) 


7.02 


Posttest: Total 
Math 


13 


48 


33.29 (67%) 


9.61 



A paired t-test for the two subtests and the total test score on the Stanford Achievement Tests: 
Grade Five resulted in a significant increase (Procedures: t= 2.64; p<.0001); (Problem Solving: 
/=4.68; pc.0001); (Total Math: /=4.10; pc.0001). 
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I Grade Seven Test Results 



Descriptions of each of the assessments used in the study and the assessment results are reported 
below: 

TABLE 8 

Learning Goals for Grade 7 Chapters 14-17 



Grade 7 Chapters 14-17 




To draw a diagram to solve problems using ratios 


To use rates, ratios, and proportions to compute unit rates and prices to solve problems 


To use tables and graphs to show rates 


To use proportions to identify the Golden Ratio 


To change ratios to percents 


To find the percent of a number 


To find what percent one number is of another number 


To find a number when the percent is known 


To identify similar figures and use scale factors to make similar figures 


To use proportions to find unknown lengths of sides of similar figures 


To use scale factors and proportions to relate area or volume of similar figures 


To use measurement and scale factors to draw similar figures 


To read and made scale drawings 


To use similar figures to measure lengths and distances indirectly 



Table 9 summarizes the Pretest and Posttest means and standard deviations for the Grade 7 Math. 

TABLE 9 

Grade 7 Results (N=20) 



Grade 7 


Math (24 test items) 




Lowest Score 


Highest Score 


Mean Score 
(% correct) 


Standard 

Deviation 


Pretest 


7 


20 


15.53 (65%) 


3.56 


Posttest 


16 


24 


21.50 (90%) 


2.21 



A paired t-test for the mean Harcourt Math Assessment showed that the scores improved 
significantly after instruction (t=8.72; p<.05). 
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Summary of Results 

• The increase in test scores on both of the nationally standardized tests, the Stanford 
Achievement Tests and on the Harcourt Math Assessment are both positive and statistically 
significant for all subtests and total test scores at both grades 2, 5, and 7. 

• The gains on both the instructional assessments and the nationally standardized tests were 
significantly different. The instructionally sensitive chapter tests showed greater 
improvement than did the national standardized test. However, the scores on the national 
standardized tests did improve statistically significantly. 

• Gains of this magnitude for such a brief period of instruction are quite remarkable 
considering that the teachers volunteered to teach the units and did not receive any extra 
training. 

• Some of the results may have shown even greater gains. However, there was a ceiling effect 
for some of the Harcourt Chapter tests. (Students scored perfect or almost perfect scores on 
the posttests thus limiting the gain scores.) 

• It is also significant that no test scores stayed the same. They all increased. In a short-term 
study of this sort it is not uncommon to note some test scores that do not increase at all. 

The percent of gains on each of the tests is shown in Table 11: 

TABLE 11 



Summary of Test Score Increases 





Percent Correct 
on Pretest 


Percent Correct 
on Posttest 


Percent 
Gain from 
Pretest to 
Posttest 


Grade/Test 




Grade 2: Harcourt Math 
Assessment 


70% 


91% 


21% 


Stanford Achievement Test 
Grade 2: Procedures 


84% 


90% 


6% 


Stanford Achievement Test 
Grade 2: Problem Solving 


74% 


84% 


10% 


Stanford Achievement Test 
Grade 2: Total Score 


78% 


87% 


9% 


Grade 5: Harcourt Math 
Assessment 


32% 


70% 


38% 


Stanford Achievement Test 
Grade 5: Procedures 


51% 


58% 


7% 


Stanford Achievement Test 
Grade 5: Problem Solving 


64% 


72% 


8% 


Stanford Achievement Test 
Grade 5: Total Score 


59% 


97% 


8% 


Grade 7: Harcourt Math 
Assessment 


65% 


90% 


25% 


Averages 


64% 


82% 


15% 
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